Networks have substantially increased the usefulness of computers such as personal computers (PC's). A local-area network (LAN) such as an Ethernet can connect several PC's together, and an Internet router or gateway on the Ethernet can allow all PC's to reach the larger Internet.
Peripherals and other devices such as keyboards and mice may be connected to an individual PC using smaller, less complex networks such as Peripheral Component Interconnect Express (PCIE) or Universal-Serial-Bus (USB). Such peripheral devices were traditionally daisy-chained together, but more recently USB hubs or PCIE switches are being used.
FIG. 1 shows a PC connecting to peripherals using a PCIE switch. PCIE switch 20 has several ports that connect to PCIE links 22. Processor host 10, such as a processor on a PC, acts as the host or root controller for all PCIE transactions. PCIE media-access controller (MAC) 12 converts commands and data from host 10 to PCIE packets that are transmitted serially over PCIE link 22 to PCIE switch 20.
Similarly, each peripheral such as video 18, disk 14, and Ethernet NIC 16 has its own PCIE MAC 12 for communicating over a PCIE link 22 to PCIE switch 20. Video 18, disk 14, and Ethernet NIC 16 act as PCIE slave devices while host 10 acts as the master. Each PCIE link 22 has two pairs of signal wires that carry full-duplex differential data in both directions. Power and ground may also be present in cables for PCIE links 22.
PCIE switch 20 switches data among ports for PCIE links 22 as required.
While PCIE is useful, the PCIE standard allows for only one host with a single address space. Connecting two or more hosts together is problematic and may required specialized software. Communication buffers such as I/O ports may be used. To connect two or more hosts together, and extension of PCIE known as Advanced Switching (AS) was developed.
FIG. 2 shows 2 hosts sharing some peripherals over a common AS switching network. Processor A in host 10 and processor B in host 40 could be separate PC's, or could be two processors in a single PC, or separate blade servers sharing a chassis. Rather than use PCIE switches, a different higher-level protocol built using the physical layer of PCIE is used, Advanced Switching (AS). AS switches 32, 34, 36 can have more than one host. AS packets include routing information to instruct each of AS switches 32, 34, 36 how to route packets among the switch's ports.
Each of host 10, 40 has a PCIE/AS bridge 30, allowing commands and data from each host to be converted to AS packets and transmitted over AS links to AS switches 32, 34, 36. AS links 220-229 can physically be PCIE cables using two differential pairs for serial-data transmission and reception. However, rather than transmit PCIE packets, higher-level AS packets may encapsulate PCIE packets for transmission over AS links 220-229 and through AS switches 32, 34, 36.
Each host can have a set of peripherals that it frequently communicates with. For example, host 40 can use video 19, disk 44, and Ethernet NIC 46, which each have a PCIE/AS bridge 30 to connect to AS switch 34 over links 229, 227, and 228, respectively. Host 10 can primarily use video 18, disk 14, and Ethernet NIC 16 through AS switches 32, 36 and AS links 222-226.
Some sharing of peripherals is possible. For example, host 40 may access data on disk 14 that is normally used by host 10. Host 40 sends an AS packet over links 220, 221, 222, 223 and through AS switches 34, 32, 36 to reach disk 14. One host could write graphics data to the other host's video in a similar way, although potential contention could occur.
However, sharing of Ethernet NIC 16 or Ethernet NIC 46 is more difficult, since each NIC typically has only one set of command and data registers. Since a large number of registers are written over a period of time to transmit an Ethernet packet, one host could over-write part of the data or commands in these registers, ruining the packet transmission for the other host. So generally Ethernet NIC's are not shared.
FIG. 3 highlights turnpool routing through AS switches. In contrast to Internet routers, AS switches 32, 34, 36 are much simpler. Rather than have routing tables that look up packet destination addresses, AS switches 32, 34, 36 read low-level routing information from AS packets. This routing information tells each of AS switches 32, 34, 36 how to route AS packets through that switch. This routing information is known as turnpool information, since the routing information tells each AS switch how many ports to rotate through to find the exit port.
For example, for AS switch 34, an AS packet is received over link 220 from host 40. Turnpool information contained in the header of this AS packet includes turnpool 38, a direction bit, and a turnpool pointer. Turnpool 38 is a field that has a turn number for each switch the packet should pass through to reach its destination (disk 14 in this example). Turnpool 38 is (5, 1, 3) in this example. Since the direction bit is 0, the direction is forward, and turnpool 38 is read forwards, starting with 5, then 1, then 3, for AS switches 34, 32, 36, respectively. The turnpool pointer is incremented after each AS switch to point to the next turn number in turnpool 38. Initially the turn pointer points to the first turn number, 5.
Each turn number in turnpool 38 indicates how may ports to skip over between the ingress port and the egress port of that AS switch. For example, the AS packet first reaches AS switch 34 over link 220 on port P0. The turn number 5 indicates that 5 ports are skipped over to find the egress port. Thus ports P1, P2, P3, P4, and P5 are skipped over, and port P6 is the egress port. The AS packet is then sent out port P6, over link 221 to AS switch 32. The turnpool pointer is decremented to point to the next turn number in turnpool 38, the second number 1.
The AS packet from link 221 is received on port P3 of AS switch 32. The middle turnpool value is 1, so one port (P4) is skipped. The egress port is P5. The turnpool pointer is decremented to point to the next turn number in turnpool 38, the first number 3. The AS packet is sent out port P5 of AS switch 32, over link 222 to switch 36.
Switch 36 receives the AS packet from link 222 on port P3. The first turn number is 3, so 3 ports (P4, P5, P6) are skipped over to find the egress port (P7). The AS packet is sent out port P7 over link 223 to disk 14.
A reply packet can easily be sent back from disk 14 to host 40 using the same turnpool. Turnpool 38 is coped into the AS header of the reply packet, and the turnpool pointer is reset and the direction bit is inverted. The direction bit of 1 indicates that turnpool 38 is read in backward order, and that ports are skipped over in counter-clockwise rather than clockwise order. Thus the reply packet can be routed back over the same route using the same turnpool 38 by inverting the direction bit.
During initialization, AS packets can be broadcast from one host or initialization device to all possible ports on all switches. Packets that reach destinations are replied to and sent back to the initialization device, which can build a table of network connections and turnpool values to reach those destinations.
FIG. 4 highlights PCIE tunneling through an AS fabric. Protocol-interface 8 (P18) is a special encoding used with AS packets when the AS packet encapsulates a PCIE packet. Data to be transmitted over the Internet or an Ethernet, such as a packet fragment, may be placed in the data payload of a PCIE packet. The PCIE packet is encapsulated by an AS header and P18 encoding is applied. The turnpool information needed to route the AS packet to the Ethernet NIC is added to the AS header, allowing the packet to pass through the AS switches in the AS fabric. This path through the AS fabric is a PI8-encoded tunnel.
The peripheral receives the AS packet, removes the P18 encoding and extracts the PCIE packet. The data payload of the AS packet contains the Ethernet packet fragment which can be combined with other fragments and sent over the Ethernet as an Ethernet packet.
FIG. 5 shows encapsulation of a PCIE packet by AS. PCIE packet 60 can carry a variety of data in data payload 62. Ethernet data such as Ethernet packet fragment 64 can be carried in PCIE data payload 62. PCIE header 58 is attached to payload 62 to form PCIE packet 60.
PCIE packet 60 is encapsulated by AS packet 50. Turnpool information including turn numbers, a direction bit, and a turnpool pointer is added as turnpool field 54. AS header 52 contains additional AS header information. A checksum or AS CRC 56 may be added for error detection. AS packet 50 is sent over the AS fabric from the host to a peripheral.
While AS switching is useful, it is desirable to share peripherals among two or more hosts on the AS fabric. A shared Ethernet NIC for use with AS networks is desirable. A shared Ethernet NIC that uses features of AS switching to more effectively and transparently share a network gateway is desirable.